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EVALUATION DEVICE, EVALUATION METHOD AND PROGRAM 

This is a continuation of Application PCT/JP02/10141, filed on September 30, 
5 2002, currently pending. 

BACKGROUND OF THE INVENTION 

Technical Field 

The present invention relates to a technology 

of evaluating usability of a device. 

10 . 

Background Arts 

A usability evaluation device and a method 
thereof, which are disclosed, e.g., in Japanese 
Patent Application Laid-Open Publication No. 2001- 

15 56771 already proposed by the present applicant, are 
know as this type of technology. This usability 
evaluation device evaluates usability of an object 
device by evaluating time till a user attains a 
target action in an object device such as ATM 

20 (Automatic Teller Machine) , a ticket issuing/selling 
machine or a cellular phone, etc.. 

These object devices are structured to 
accomplish one object by performing, as their 
functions, a series of operations such as drawing 

25 cash, depositing the cash, purchasing a ticket, 

giving a phone call, receiving the phone call and so 
forth. Therefore, the usability evaluation device 
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finally accomplished by sequentially measuring the 
time till each of the operations is completed. 

Depending on the object device, however, the 
completion of the operation can not necessarily be 
judged distinctively. Namely, the object device of 
which the operation continuously proceeds, might have 
a case of being unable to distinctively identify the 
completion of the operation. Further, depending on 
an operation tool, there might be case where the 
completion of the operation can not necessarily be 
recognized distinctively. 

For example, in the case of setting a 
temperature, time, a volume, tuning in on TV and 
radio, etc., by use of an up/down switch, there might 
15 be often an excess over a target position. This is a 
case in which if the temperature is desired to be set 
at 25 degrees, the temperature is changed up further 
to 26 degrees from 24 degrees and 25 degrees. In 
this case, after the target setting has been 
temporarily done, the setting further deviates. The 
prior arts did not have any contrivance about judging 
the case given above. 

Moreover, if the object device to be evaluated 
includes a wide variety of operating components, the 
25 evaluation based on operations other than the 

operation upon a detection object can not be made 
simply by detecting only completion of a series of 



20 
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operations . 

Moreover, according to the technology disclosed 
in the above Publication, the usability of the object 
device is evaluated by obtaining a ratio of operation 
time of a well-experienced user who operates the 
object device to operation time of a general user. 
Therefore, if the general user has a variety of 
attributes, those attributes can not be reflected in 
the evaluation of the object device. 



SUMMARY OF THE INVENTION 

The present invention was devised in view of 

the problems inherent in the prior arts described 

above. Namely, it is an object of the present 
15 invention to provide a technology capable of 

objectively evaluating usability of a device 

including a variety of operating components . 

It is another object of the present invention 

to provide a technology capable of reflecting a 
20 variety of user's attributes in the evaluation of the 

device . 

The present invention adopts the following 
means in order to accomplish the above objects. 
Namely, the present invention is an evaluation device 
25 for evaluating usability of an object device on the 
basis of an operation by an examinee upon the object 
device, comprising means for defining an initial 



status of the object device, means for defining a 
target status after operating the object device, 
operation identifying means for identifying an 
examinee's operation upon the object device, status 
identifying means for identifying an operation status 
of the object device by the examinee's operation, 
judging means for judging whether the operation 
status is coincident with the target status or not, 
and time measuring means for measuring elapsed time 
till the operation status reaches the target status 
from the initial status. 

Thus, according to the present invention, the 
operation status of the object device based on the 
examinee's operation is compared with the target 
status, and the time till the operations status 
reaches the target status is measured. Therefore, 
the usability of the object device is judged based on 
the time. On this occasion, an unnecessary operation 
other than the operation for reaching the target 
status is analyzed, and this analysis can be 
reflected in evaluation the object device. Further, 
the means for defining the target status defines the 
target status without any restriction, and the object 
device can be evaluated in a variety of procedures. 

Preferably, the target status may contain a 
plurality of transition statuses, the judging means 
may sequentially judge whether the operation status 
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is coincident with each of the plurality of 
transition statuses, and the time measuring means may 
measure each elapsed time till each of the transition 
status is reached. 
5 As described above , according to the present 

invention, the object device can be evaluated based 
on the elapsed time in each step till the target 
status is finally reached through transitions of the 
plurality of transition statuses. 

10 Preferably, the operation identifying means may 

include a detection unit for detecting the operation 
upon the object device, and a notifying unit for 
notifying of the detected operation. 

Preferably, the operation identifying means may 

15 include means for receiving a signal generated 

corresponding to the operation upon the object device, 
and the status identifying means may identify the 
operation status by referring to a signal definition 
part in which a change in the status of the object 

20 device that corresponds to the signal is defined. 

Preferably, the evaluation device may further 
comprise means for referring to time when receiving 
the signal, and means for recording the time when 
receiving the signal together with the operation 

25 status identified by the received signal. Thus, when 
receiving the signal, the operation status of the 
device may be identified and recorded. 
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Preferably, the operation identifying means may 
include means for reading a recording part recorded 
with the signal generated corresponding to the 
operation upon the object device and with generation 
5 time of this signal, the status identifying means may 
identify the operation status by referring to the 
signal definition part in which the change in the 
status of the object device that corresponds to the 
signal is defined, and the time measuring means may 

10 include means for calculating the elapsed time on the 
basis of the generation time read from the recording 
part. As explained above, at first, the type of the 
generated signal and the generation time are recorded, 
and the status of the device may separately be 

15 identified. 

Preferably, the judging means may judge that 
the operation ' status is coincident with the target 
status when the operation status is continuously in 
the target status during a predetermined period. 

20 Thus, according to the present invention, if the 

coincident status continues during the predetermined 
period, this is judged to be coincident. Hence, if 
the examinee's operation exceeds a target value, it 
is possible to prevent a misjudgment that the target 

25 value has been set. 

Preferably, the evaluation device may further 
comprise means for registering attributes of the 
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examinee operating the object device, means for 
adding up the elapsed time for every attribute of the 
examinee, and means for calculating a ratio of an 
added-up result about examinees having a first 
5 attribute to an added-up result about examinees 
having a second attribute. 

Thus, according to the present invention., the 
measurement results when the individual examinees 
operate the object device can be classified and added 

10 up in accordance with the attributes of the examinees. 
This contrivance enables the evaluation to contain a 
relationship between the attributes of the examinee 
and the usability of the device. . 

Further, the present invention may also be a 

15 method by which any one of the functions described 

above is actualized. Moreover, the present invention 
may also be a program for making a computer actualize 
any one of the aforementioned functions. Still 
further, the present invention may be a readable-by- 

20 computer recording medium recorded with such a 
program . 

Herein, the readable-by-computer recording 
medium connotes recording mediums capable of storing 
information such as data, programs, etc. electrically, 
25 magnetically, optically and mechanically or by 

chemical action, which can be read by the computer. 
What is demountable out of the computer among those 
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recording mediums may be, e.g., a flexible disk, a 
magneto-optic disk, a CD-ROM, a CD-R/W, a DVD, a DAT, 
an 8 mm tape, a memory card, etc.. 

Further, a hard disk, a ROM (Read Only Memory) 
5 and so on are given as recording mediums fixed into 
the computer . 

As explained above, according to the present 
invention, the usability of the device including the 
variety of operating components can be objectively 
10 evaluated. Furthermore, according to the present 

invention, the variety of attributes of the user can 
be reflected in the evaluation of the device. 

BRIEF DESCRIPTION OF THE DRAWINGS 
15 FIG. 1 is a conceptual diagram of a usability 

evaluation device in one embodiment of the present 
invention; 

FIG. 2 is a view of a system structure of the 
usability evaluation device; 
20 FIG. 3 shows an example of a control definition 

screen ; 

FIG. 4 shows an example of an operation step 
definition screen ; 

FIG. 5 shows an example of an examinee 
25 information definition screen; 

FIG. 6 shows an example of the examinee 
information definition screen; 
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FIG. 7 shows an example of the examinee 
information definition screen; 

FIG. 8 shows an example of a log data display 
screen ; 

5 FIG. 9 shows an example of a comparison and 

verification based on an attribute ratio between the 
examinees ; 

FIG. 10 shows an example of the comparison and 
the verification based on the attribute ratio between 
10 the examinees; 

FIG. 11 is a flowchart showing a log 
information acquisition process; 

FIG. 12 is a flowchart showing a log data 
analyzing process; and 
15 FIG. 13 is a flowchart showing a correctness 

judging process. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

A usability evaluation device according to a 
20 preferred embodiment of the present invention will 

hereinafter be described with reference to the 

drawings in FIGS. 1 through 13. 

FIG. 1 is a conceptual diagram of the usability 

evaluation device in the present embodiment. FIG. 2 
25 is a view of a system structure of the usability 

evaluation device. FIGS. 3 through 7 show examples 

of a definition screen for defining various 



categories of information in this usability 
evaluation device. FIG. 8 shows an example of a log 
data display screen of the usability evaluation 
device. FIGS. 9 and 10 show screens for a 
comparing/verifying operation based on attributes of 
an examinee. FIG. 11 is a flowchart showing a log 
information acquisition process executed by a CPU 1 
in FIG. 2. FIG. 12 is a flowchart showing a log data 
analyzing process. FIG. 13 is a flowchart showing a 
correctness judging process. 
<Concept of Device> 

FIG. 1 shows the conceptual diagram of the 
present usability evaluation device. The present 
usability evaluation device is constructed of an 
operation detecting module for detecting user's 
(examinee's) operations (events) on a variety of 
devices and for generating a predetermined key code 
(a bit-string of 1-byte) corresponding to the 
operation, and of an evaluation computer for 
receiving the key code from the operation detecting 
module, recording the key code as log data and 
analyzing the data. 

For evaluation by the usability evaluation 
device, to begin with, the operation detecting module 
is attached to an object device (which may be defined 
herein either hardwarewise or sof twarewise) to be 
evaluated, or alternatively, the operation detecting 



module is installed into the object device. The 
evaluation object device is exemplified such as a 
product for providing a function to the user in 
response to the user's operation, a hardware model of 
this type of product, GUI (Graphical User Interface) 
on a computer, or a Web page browsed through Browser, 
and so on. 

As for the product and the hardware model of 
the product, the operation detecting module is 
attached to a switch unit of the product. The 
operation detecting module is exemplified such as a 
contact switch for detecting that the user presses a 
push button, an area sensor for detecting a user's 
operation upon a predetermined area, a touch panel 
and so forth. 

One of these operation detecting modules may be' 
connected directly to an input port of the evaluation 
computer. In this case, a driver program on the 
evaluation computer converts a signal from the 
operation detecting module into a predetermined key 
code . 

Further, one of those operation detecting 
modules may also be connected to a computer dedicated 
to the detection, whereby the key code may be 
generated. In this case, the key code generated may 
be transferred to the evaluation computer from the 
computer dedicated to the detection. 



- 12 - 



Moreover, for example, if the object device is 
operated through infrared-ray communications by a 
remote controller, the operation detecting module is 
a signal converting device for receiving and 
5 converting infrared-rays from the remote controller 
into a key code (a predetermined bit-pattern) 
corresponding thereto and inputting the key code to 
the evaluation computer. 

Furthermore, if the object device is a computer 
10 including the GUI, the operation detecting module is 
a pseudo program of simulation for capturing an event 
from the GUI and outputting a key code corresponding 
to this event. This type of pseudo program is 
defined corresponding to, for instance, GUI buttons 
15 and control and may be executed as replaced with 
functions (callback routine) . 

Still further, if the evaluation object is a 
Web page, the operation detecting module is a 
dedicated piece of Browser for generating a key code 
20 when, for example, a button and a link of the Web 

page are manipulated. After the operation detecting 
module described above has been installed into the 
object device or the object computer, the examinee 
operates the object device. The key code 
25 corresponding to the element manipulated is thereby 
generated, whereby the object device is to be 
evaluated . 
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For evaluating the object device, information 
(test procedures) for evaluation is preset in the 
evaluation computer. The evaluation information is, 
e.g., attribute information (a distinction of sex, an 
age, a using experience, etc.) of the examinee, an 
evaluation object (information such as a name of a 
test machine, etc.), correctness information serving 
as a criterion for evaluation, and so on. 

Herein, the correctness information is a series 
of operation steps on the object device. These 
operation steps are exemplified such as setting a 
temperature of a car air-conditioner at 20 degrees, 
setting a CD track to No . 5 , subsequently opening a 
window of a rear seat, and so forth. These test 
procedures may, however, be predetermined, and, after 
acquiring the log data, the correctness information 
may be set in the evaluation computer when analyzing 
the log data. 

Obtained next are the log data when the 
examinee performs the operation in accordance with 
the test procedures . The log data can be obtained 
from the key code generated by the operation 
detecting module and time (corresponding to operation 
time) when each key code is generated. 

The evaluation computer analyzes and organizes 
the obtained log data into a graph. At this time,, 
there are executed a comparison between a general 
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user and a well-experienced user, a comparison 
between a group of the examinees having the 
predetermined attributes and a group of other types 
of examinees, and so forth. 
5 <System Structure> 

FIG. 2 is the view showing an structure in 
which a usability evaluation device 11 in the 
embodiment of the present invention is combined with ' 
a hardware model 10 in order to evaluate the 

10 usability of the hardware model 10 of a car, an ATM , 
a cellular phone and so forth. As shown in FIG. 2, 
the usability evaluation device 11 includes a CPU 1 
for controlling the device, a sensor 2 (corresponding 
to an operation detecting module) connected to an I/O 

15 port of the CPU 1 and generating operation signals 
when the user manipulates switches of the hardware 
model 10, a hard disk device 3 for recording the 
user's operation taken in from the sensor 2, a 
display device 4 for displaying a result of the 

20 evaluation, a keyboard 5, a video camera 6 for 

capturing an operation image of the user's operation, 
a VTR 7 for recording the captured operation image, 
and an image display device 8 for displaying the 
recorded image. 

25 The sensor 2 is the contact switch provided in 

superposition on the push button 12 of the hardware 
model, the infrared-ray area sensor for detecting 
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intrusion of the hand and the fingers into the 
predetermined area, and so on. When the examinee 
presses the push button, the contact switch detects 
this pressing operation. Further, when the examinee 
5 moves his or her hand in close proximity to the 

switch in a predetermined position, the area sensor 
detects the intrusion of the hand. The sensor 2 
generates a detection signal of the operation 
described above, and transmits the detection signal 
10 to the CPU 1 via the I/O port. 

The CPU 1 executes a control program lb 
retained on an unillustrated memory, thereby 
controlling the device. Further, the CPU 1 converts 
an address of the I/O port or a position signal from 
15 the area sensor into a key code corresponding thereto. 
Moreover, the CPU 1 measures time when 
detecting each operation signal with the aide of a 
built-in timer la (corresponding to a completion time 
measuring unit) . The control program lb executed by 
20 the CPU 1 stores a result of these measurements 

(types of the detected operations, and detection time 
of each of the operations) as pieces of log data 
(which are written as collected data 3 in FIG. 2) . 
Further, the CPU 1 separately executes an 
25 analysis program, thereby analyzing and evaluating 
the collected log data. Then, the analysis program 
has a result of the evaluation displayed the display 



device 4 . 

The video camera 6 captures an image of the 
user's operation upon the object system. The VTR 7 
records the image captured by the video camera 6 . 

The image display device 8 displays the image, 
of how the user operates the object system, which 
image has been recorded by the VTR 7. This image, 
though not used directly for the evaluation in the 
present usability evaluation device, is 
supplementally utilized for verifying validity of the 
result of the evaluation and for adding a subjective 
analysis to the result of the evaluation by the 
present usability evaluation device. 

<Screen Layout> 

FIGS. 3 through 7 show the examples of the 
definition screen for defining the various categories 
of information in the present usability evaluation 
device. FIG. 3 shows the example of a control 
definition screen. On the control definition screen, 
there is defined a piece of information (this piece 
of information is called control) inputted from each 
of the operation units of the hardware model 10 to be 
evaluated. 

On the control definition screen, there are set 
a name of the control ((1) in FIG. 3), a type of the 
control ( (2) in FIG. 3) and a corresponding key code 
and data of every control ((3) in FIG. 3). 



The name of the control is a name of a function 
of the operation unit concerned. For example, in the 
case of the operation unit for setting a temperature, 
a name [Temperature] is set. 

The type of the control is a type of the switch 
structuring the operation unit. In the usability 
evaluation device according to the present embodiment, 
the following six types are provided. 

(1) ON/ OFF 

This is a switch that is switched ON/ OFF 
repeatedly each time this switch is pressed as a 
power source switch is. If the control type is 
ON/OFF, only a key code corresponding thereto is set, 
and other pieces of data are not defined. 

(2) UP /DOWN 

This is a switch in which a value is 
incremented or decremented each time the switch is 
pressed as in the case of setting the temperature. 
If the control type is UP /DOWN , as indicated by (3) 
in FIG. 3, a key code corresponding to UP, a key code 
corresponding to DOWN, a value range and an 
increment/decrement of the value are set as pieces of 
data. Note that the increment/decrement of the value 
is an increment quantity for the UP operation and a 
decrement quantity for the DOWN operation. 

(3) CYCLIC 

This is a switch in which each time one switch 
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like a changeover switch of an air blow port of, e.g., 
the car air-conditioner is pressed, a status changes 
such as [Window] => [Front] => [Underfoot] => 
[Window] . If the control type is CYCLIC, only a key 
5 code corresponding the next status to be transited is 
set, and other pieces of data are not defined. 

(4) GROUP 

This is a switch in which to select such a 
single status that other switches are canceled when 
10 pressing one switch. If the control type is GROUP, a 
key code corresponding to each status is defined as 
the data. 

(5) OFF-GROUP 

This is, though similar to GROUP, a switch for 
15 allowing a state where none of the statuses are 
selected. If the control type is OFF-GROUP, key 
codes corresponding to the respective statuses and to 
the non-status case are likewise defined as the data. 

(6) RESET 

20 This is a switch for changing (for setting a 

RESET status) the statuses of other switches. 

FIG. 4 shows the example of the definition 
screen for the operation steps. The usability 
evaluation device evaluates the device on the basis 

25 of a combination of one or more operation steps (the 
operation step may also be referred to as a task) . 
The operation unit (control) operated in each 
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operation step is defined on the operation step 
definition screen in FIG. 4. 

As shown in FIG. 4, the operation step 
definition screen contains a title box ((1) in FIG. 
5 4), status setting boxes ((2) in FIG. 4), operation 
selecting checkboxes ((3) in FIG. 4) and an operation 
step display box ((4) in FIG. 4) . 

A name of the operation step (task) defined at 
the present is designated in the title box. Herein, 

10 if the name of the operation step is an [initial 

status] , the initial status of the device is defined. 

One or more categories of the control operated 
in this operation step are designated in the 
operation selecting checkboxes ((3) in FIG. 4). In 

15 FIG. 4, for example, the control categories such as a 
"temperature", an "audio", a "CD track", an 
"inside/outside air", etc. are designated by 
checkmarks. Thus, in the usability evaluation device, 
the plurality of control categories can be designated 

20 for the single operation step. 

Statuses that should be set in the present 
operation step for the respective control categories 
selected in the operation selecting checkboxes ( (3) in 
FIG. 4), are specified in the status setting boxes 

25 ((2) in FIG. 4). For instance, 18 degrees is entered 
in the "temperature", AM/FM (a status to which the 
radio is tuned in) is entered in the "audio", 1 is 
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entered in the "CD track", the inside air is entered 
in the "inside/outside air", and so forth. 

Thus, after setting one operation step, a new 
operation step is added by pressing a "new step 
5 insertion button" ((5) in FIG. 4) or a "new step 

adding button" ((6) in FIG. 4), and the definitions 
in the title box ((1) in FIG. 4), the operation 
selecting checkboxes ((3) in FIG. 4) and the status 
setting boxes ((2) in FIG. 4), are repeated. A test 
10 sequence structured of a series of operation steps 

can be thereby defined. This test sequence is stored 
as the correctness information on the hard disk 
device 3 . 

FIGS. 5 through 7 show the examples of the 
15 examinee information definition screen on which the 
user's (examinee's) attributes are defined. In the 
usability evaluation device, as shown in FIG. 5, a 
"log ID", an "ID", a "name", "reading"., "NE (a 
distinction between Novice and Expert) , a 
20 "distinction of sex", an "address" and a "telephone 
number", are prepared as the examinee information. 

Further, in the usability evaluation device, 
the user is able to define attributes other than the 
attributes prepared beforehand in the device. Namely, 
25 as shown in FIG. 6, the user may click an "entry 

adding button" ((1) in FIG. 6), then input a name of 
the entry ((2) in FIG. 6) and click an "OK button" 



((3) in FIG. 6) . Fig. 7 shows an example of the user 
definition entries. Added in FIG. 7 are entries such 
as a "job", an "annual income", "years of driving 
experience", "mileage per month", "possession or non- 
possession of car navigation system", and so on. 

FIG. 8 shows an example of a log data display 
screen for displaying the obtained log data. As 
shown in FIG. 8, the log stored on the usability 
evaluation device consists of a "step", a "code (a 
simple notation of the key code) , a "key name 
acguisition time" and respective control statuses 
(such as a "temperature", "audio", a "CD track", 
etc . ) . 

For instance, in FIG. 8, after starting the 
test (which indicates a row where the start key is 
specified in the "key name") , a temperature setting 
operation specified by a key code Q is repeated, and 
it can be understood that the set temperature rises 
such as 18 degrees, 18.5 degrees, 19 degrees, 19.5 
degrees and so forth. 

FIGS. 9 and 10 show examples of performing a 
comparison and verification based on an attribute 
ratio- among the examinees . The attribute ratio among 
the examinees is a ratio of measurement data among 
the examinees each having different attributes, 
wherein this ratio is obtained by performing a 
predetermined evaluation test repeatedly by a 
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plurality of examinees each having the different 
attributes, and adding up logs when operated by the 
examinees exhibiting the same attributes. The 
measurement data is, for example, average time 
required for executing a predetermined operation step. 

For instance, in FIG. 9, NE (the distinction 
between Novice and Expert) and the years of driving 
experience are selected as attributes. Then, in FIG. 
10, logs of the examinees of a first group as Experts 
each having ten or more years of their driving 
experiences are added up and indicated by a solid 
line graph. Further, logs of the examinees of a 
second group as Novices each having three to five 
years of their driving experiences are added up and 
indicated by a dotted line graph. Moreover, a rate 
of the operation time in each operation step is 
indicated by a bar graph. 

<Operation> 

FIG. 11 shows a flowchart of a log data 
acquisition process in the usability evaluation 
device. This process is actualized by an application 
program executed by the CPU 1 of the usability 
evaluation device . 

When starting up the log data acquisition 
process, the usability evaluation device, at first, 
receives the key code (SI). Then, the usability 
evaluation device reads the key code and refers to 
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the definition of the control defined corresponding 
to this key code (see FIG. 3) , thereby judging a type 
of the operation switch. 

Subsequently, the usability evaluation device 
judges whether the type of the operation switch is 
ON/OFF or not (S2) . If the type of the operation 
switch corresponding to the key code is ON/ OFF , the 
usability evaluation device replaces and sets the 
status (ON/OFF) of the control that corresponds to 
the key code, and outputs the control status to the 
log (S3) . Thereafter, the usability evaluation 
device returns the control to SI. 

Further, whereas if the type of the operation 
switch corresponding to the key code is not ON/ OFF , 
the usability evaluation device judges whether the 
type of the operation switch is UP /DOWN or not (S4) . 
If the type of the operation switch corresponding to 
the key code is UP /DOWN , the usability evaluation 
device reads an increment/decrement value 
corresponding to this key code from the control 
definition. Then, the usability evaluation device 
adds the increment/decrement value to the present 
control value in the case of UP and, in the case of 
DOWN, subtracts the increment/decrement value 
therefrom, and outputs a result to the log (which is 
described as integrating in S5 in FIG. 11) . 
Thereafter, the usability evaluation device returns 
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the control SI . 

Moreover, whereas if the type of the operation 
switch corresponding to the key code is not UP/DOWN , 
the usability evaluation device judges whether the 
5 type of the operation switch is CYCLIC or not (S6) . 
If the type of the operation switch corresponding to 
the key code is CYCLIC, the usability evaluation 
device advances the control status corresponding to 
the key code to a next status, and outputs this next 

10 status to the log (S7) . Thereafter, the usability 
evaluation device returns the control to SI. 

Further, whereas if the type of the operation 
switch corresponding to the key code is not CYCLIC, 
the usability evaluation device judges whether or not 

15 the type of the operation switch is GROUP or OFF- 
GROUP (S8) . If the type of the operation switch 
corresponding to the key code is GROUP or OFF-GROUP, 
the usability evaluation device reads a status 
corresponding to the key code from the control 

20 definitions, and outputs this status to the log (S9) . 
Thereafter, the usability evaluation device returns 
the control to SI . 

Furthermore, if the type of the operation 
switch corresponding to the key code is neither GROUP 

25 nor OFF-GROUP, the usability evaluation device judges 
whether the type of the operation switch is RESET or 
not (SA) . If the type of the operation corresponding 
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to the key code is RESET, the usability evaluation 
device resets a value of the control corresponding to 
the key code , and outputs a result of resetting to 
the log (SB) . Thereafter, the usability evaluation 
5 device returns the control to SI. 

Moreover, whereas if the type of the operation 
switch corresponding to the key code is not RESET, 
the usability evaluation device judges whether the 
log data acquisition process is ended or not (SC) . 

10 If the end of the log data acquisition process is 
designated, the usability evaluation device 
terminates the process. Further, whereas if the end 
of the log data acquisition process is not designated, 
the usability evaluation device returns the control 

15 to SI . 

Note that the judgment of whether the log data 
acquisition process is ended or not, is made also by 
monitoring whether an end button of an unillustrated 
log data acquisition window is pressed or not, 
20 independently of the processing flow in FIG. 11. 

FIG. 12 shows a flowchart of a log data 
analyzing process. In this process, the usability 
evaluation device, to start with, reads information 
of a next step from the correctness information 
25 (which is set in FIG. 4) (S20) . 

Next, the usability evaluation device reads a 
next record of the log (S21) . Then, the usability 
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evaluation device executes a correctness judging 
process. Subsequently, the usability evaluation 
device judges whether a result of the correctness 
judging process is correct or not (S22 , S23) . 
5 If the result of the correctness judging 

process is correct, the usability evaluation device 
calculates elapsed time till the present step is 
completed, and records the calculated time in an 
analysis result of the operation time (S25) . 

10 Next, the usability evaluation device judges 

whether or not a next step is defined in the 
correctness information (S26) . If the next step 
exists in the correctness information, the usability 
evaluation device returns the control to S20, and 

15 starts analyzing the next step. 

Further, whereas if the next step does not 
exist in the correctness information, this implies 
that all the operation steps have been finished, and 
hence the usability evaluation device displays a 

20 graph of the analysis result (S27) . Thereafter, the 
usability evaluation device terminates the process in 
accordance with, e.g., an instruction of the operator. 

While on the other hand, if the result of the 
correctness judging process is incorrect, the 

25 usability evaluation device judges whether or not the 
log contains a next record (S24) . If the log 
contains the next record, the usability evaluation 
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device returns the control to S21. 

Furthermore, whereas if the log does not 
contain the next record, the usability evaluation 
device advances the control to S27 (S27) . A graph is 
5 thereby displayed in a status where all the 
operations are not yet completed. 

Note that when displaying the graph in S27, for 
example, data of an examinee having a specified 
attribute 1 (for instance, 10 or longer years of 

10 driving experience of the car) and data of an 

examinee having a specified attribute 2 (for example, 
3 to 5 years of driving experience of the car) , are 
respectively added up, and a ratio between the 
respective added results (between, e.g., average 

15 values of the operation time of the individual steps) 
may also be displayed. 

FIG. 13 shows details of the correctness 
judging process (S22 in FIG. 12). In this process, 
the usability evaluation device compares a status in 

20 the present record of the log with a status in the 
present step of the correctness information (S221) . 
Then, the usability evaluation device judges whether 
or not the status in the present record of the log is 
coincident with the status in the present step of the 

25 correctness information (S222) . 

If these statuses are coincident with each 
other, the usability evaluation device judges whether 
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or not correct judgment time elapses after those 
statuses have become coincident (S223) . Here, the 
correct judgment time is a predetermined period of 
time during which the correctness information is 
5 confirmed to be continuously coincident with the log 
in the comparison between the correctness information 
and the log. 

If the correct judgment time elapses, the 
usability evaluation device makes a judgment of being 
10 correct, and sets a result of this judgment (S224) . 
Thereafter, the usability evaluation device 
terminates the correctness judging process. 

While on the other hand, if judged to be 
uncoincident in S222, or if the correct judgment time 
15 does not elapse in S223, the usability evaluation 

device clears the result of the correctness judgment 
(S225) . Thereafter, the usability evaluation device 
finishes the process. 



evaluation device in the embodiment, the operator 
desiring to evaluate the usability of the device can 
organize the operation steps of the desired usability 
evaluation as the operator intends by setting the 
25 correctness information on the screen in FIG. 4. It 
is therefore possible to flexibly evaluate the 
usability of the device including, e.g., a variety of 



<Effects of Embodiment> 



20 



As discussed above, according to the usability 
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operation appliances such as audio products and so on 
as the car, etc. has. 

Moreover, according to the present usability 
evaluation device , when the correctness information 
5 is continuously coincident with the log during the 
predetermined period of time in the comparison 
between the correctness information and the log (the 
result of the examinee's operation), it is judged 
that these two elements become coincident. Hence, 

10 even in the case of a mis-operation of exceeding the 
target set value in setting, e.g., a volume or the 
UP/DOWN switch, etc., a mis judgment of being correct 
can be avoided, and it is feasible to judge the 
coincidence of the correctness information with the 

15 log more surely. 

Moreover, the present usability evaluation 
device is capable of not simply comparing the well- 
experienced user with the examinee (the general user) 
other than the well-experienced user but adding up 

20 and comparing the results of the operations on the 

basis of the variety of attributes of the examinees . 
<Modif ied Example s> 

In the embodiment discussed above, the data 
temporarily stored as the log in the log data 
25 acquisition process are analyzed in the log data 

analyzing process. As a substitute for this, however, 
the received key code is directly analyzed in 
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parallel with the log data acquisition process, and 
the correctness judgment may also be made. 

In the embodiment discussed above, the key code 
is temporarily generated based on the detection 
5 signal outputted by the sensor 2, and the operation 
of the user is identified from the key code and the 
control definition. The embodiment of the present 
invention is not, however, limited to these 
procedures. For example, the operation (the switch, 

10 etc. to be operated) of the examinee may be 

identified directly from an address, etc. of an input 
port to which the detection signal is inputted 
without generating the key code. 

Further, in the embodiment discussed above, the 

15 control corresponding to the key code is identified, 
and the change in the control status is recorded in 
the log data. The embodiment of the present 
invention is not, however, confined to these 
procedures . 

20 For example, the key code and the key code 

generation time may be recorded intactly in the log 
data. Then, when analyzing the log data, the control 
corresponding to the key code may be obtained. 
Namely, in the embodiment of the present invention, 

25 the operation of the examinee and the operation time 
thereof may finally be identified. Accordingly, as 
far as those final results are to be acquired, in 



- 31 - 



recording in the in-process log or in analyzing the 
log, there may be implemented such a procedure change 
as to shift the process that is on the execution on 
one side to the other side. 

5 

<Industrial Applicability> 

The present invention can be applied to the 
usability evaluations of the devices for providing 
the functions by the operation of the user, such as 
10 information appliances, electric products, cars, 
ships, airplanes, machine tools, communication 
appliances, etc. . 



